Detection of story reader progress for pre-caching special effects

ABSTRACT

The disclosure provides technology for enhancing the ability of a computing device to conserve computing resources while providing special effects that are aligned with a text source being read aloud. An example method includes identifying effects data for a text source, wherein the effects data correlates a physical effect with an audible action of a user; receiving audio data comprising a plurality of audible actions, wherein at least one of the plurality of audible actions comprises a spoken word of the text source; determining, by a processing device, a triggering condition based on the effects data and the text source; responsive to satisfying the triggering condition, loading content for the physical effect, wherein the triggering condition is satisfied prior to the occurrence of the audible action; and providing the physical effect to modify an environment of the user.

TECHNICAL FIELD

This disclosure relates to the field of virtual assistants and, inparticular, to enhancing the ability of a virtual assistant to pre-cachespecial effects for a text source that is being read aloud.

BACKGROUND

The capabilities and usage of virtual assistants are expanding rapidly.Traditional virtual assistants include some form of computer humaninterface that enables a human to interact with the virtual assistantand cause the virtual assistant to perform a task or service. Virtualassistants often record and understand human speech and can respond bysynthesizing a reply. The virtual assistants may be initiated inresponse to a touch or gesture based command or may continuously analyzeits environment to detect a spoken command. When the command isdetected, the virtual assistant may respond or perform one or moreactions.

SUMMARY

The following is a simplified summary of the disclosure in order toprovide a basic understanding of some aspects of the disclosure. Thissummary is not an extensive overview of the disclosure. It is intendedto neither identify key or critical elements of the disclosure nordelineate any scope of the particular embodiments of the disclosure orany scope of the claims. Its sole purpose is to present some concepts ofthe disclosure in a simplified form as a prelude to the more detaileddescription that is presented later.

According to a first aspect of the present disclosure is a methodcomprising: identifying effects data for a text source, wherein theeffects data correlates a physical effect with an audible action of auser; receiving audio data comprising a plurality of audible actions,wherein at least one of the plurality of audible actions comprises aspoken word of the text source; determining, by a processing device, atriggering condition based on the effects data and the text source;responsive to satisfying the triggering condition, loading content forthe physical effect, wherein the triggering condition is satisfied priorto the occurrence of the audible action; and providing the physicaleffect to modify an environment of the user.

Optionally, determining the triggering condition may comprisedetermining the physical effect is associated with a first location inthe text source. Determining the triggering condition may compriseselecting a second location in the text source that is before the firstlocation. The selecting may be based on a reading speed and a load timeassociated with the physical effect. Determining the triggeringcondition may comprise setting the triggering condition to correspondwith the second location in the text source. The second location in thetext source may be associated with at least one of a particular instanceof a word, a paragraph, a page, or a chapter of the text source. Thedetermining of the triggering condition may comprise calculating aduration of time to load the content based on an amount of the contentfor the physical effect and an amount of available computing resources.The determining of the triggering condition may comprise determining atime in the future that the audible action will occur. The determiningof the triggering condition may comprise identifying a time to initiatethe loading based on the calculated duration and the determined time ofthe audible action. The determining of the triggering condition maycomprise initiating the loading of the content at or before theidentified time. The determining of the time may comprise calculatingthe time in the future based on a reading speed and a current readinglocation in the text source. The effects data may indicate the physicaleffect. The effects data may indicate a location in the text source thatrelates to the audible action. The location may indicate a word,paragraph, or a page of the text source. The audible action may comprisea spoken word of the text source. The physical effect may comprise anacoustic effect corresponding to the spoken word. The audible action maycomprise a page turn. The physical effect may comprise a modification toan acoustic effect, an optical effect, or a haptic effect.

According to a second aspect of the present disclosure is a systemcomprising a processing device. The processing device is configured to:identify effects data for a text source, wherein the effects datacorrelates a physical effect with an audible action of a user; receiveaudio data comprising a plurality of audible actions, wherein at leastone of the plurality of audible actions comprises a spoken word of thetext source; determine a triggering condition based on the effects dataand the text source; responsive to satisfying the triggering condition,load content for the physical effect, wherein the triggering conditionis satisfied prior to the occurrence of the audible action; and providethe physical effect to modify an environment of the user.

The system may comprise data storage. The system may comprise acommunications system for communicating over a network, such as a localarea and/or wide area network. The system may be, may be comprised in ormay be configured to implement a virtual assistant. The system may beconfigured to implement the method of the first aspect.

According to a third aspect of the present disclosure is a computerprogram product configured such that, when processed by a processingdevice, the computer program product causes the processing device toperform the method of the first aspect.

The individual features and/or combinations of features defined above inaccordance with any aspect of the present disclosure or below inrelation to any specific embodiment may be utilized, either separatelyand individually, alone or in combination with any other definedfeature, in any other aspect or embodiment. Furthermore, the presentdisclosure is intended to cover apparatus configured to perform anyfeature described herein in relation to a method and/or a method ofusing or producing, using or manufacturing any apparatus featuredescribed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1 illustrates an example environment with one or more computingdevices in accordance with an implementation of the disclosure.

FIG. 2 is a block diagram illustrating an example computing device withcomponents and modules for comparing phoneme data derived from userinput and phoneme data derived from a text source in accordance with animplementation of the disclosure.

FIG. 3 is a block diagram illustrating an example computing device withcomponents and modules for identifying a location in the text sourcebased on the audio input of a user in accordance with an implementationof the disclosure.

FIG. 4 is a block diagram illustrating an example computing device withcomponents and modules for providing physical effects to enhance anexperience of one or more users in accordance with an implementation ofthe disclosure.

FIG. 5 is a flow diagram illustrating an example of method in accordancewith an implementation of the disclosure.

FIG. 6 is a flow diagram illustrating an example of method in accordancewith an implementation of the disclosure.

FIG. 7 is a flow diagram illustrating an example of method in accordancewith an implementation of the disclosure.

FIG. 8 is a flow diagram illustrating an example of method in accordancewith an implementation of the disclosure.

FIG. 9 is a block diagram illustrating another example of a computingdevice in accordance with an implementation of the disclosure.

These drawings may be better understood when observed in connection withthe following detailed description.

DETAILED DESCRIPTION

Modern computing devices often provide features to detect and understandhuman speech. The features may be associated with a virtual assistantthat may be accessible via a computing device that is resourceconstrained, such as smart speaker, mobile phone, smart watch, or otheruser device. The computing device may be associated with a microphonethat can record the human speech and may use a combination of local andremote computing resources to analyze the human speech. Analyzing speechis typically a resource intensive operation and the computing device maybe configured to perform some of the processing local to a computingdevice and has some of processing performed remotely at a server or viaa cloud service. Many virtual assistants use some form of a remotespeech recognition service that takes audio data as input and convertsthe audio data into text that is returned to the computing device.

Many technical problems arise when attempting to use traditional virtualassistant features to provide sound effects that are syncronized withthe spoken content of a text source. Some of the problems arise becausea traditional virtual assistant performs speech recognition to translateaudio into text and then performs a comparison based on the text. Thespeech recognition typically involves an acoustic step that translatesthe audio into phonemes and a language step that translates the phonemesinto text. The language step often waits for subsequent spoken words toestablish context before translating a spoken word into text. Thelanguage step introduces a time delay and consumes additional computingresources on a computing device that may be resource constrained. Thedelay may be further compounded because the sound effects may be largeaudio files that are downloaded from a remote data source. A traditionalapproach may involve downloading the sound effects in response todetecting a spoken word but the delay may result in the special effectbeing provided long after the word is spoken. Another approach mayinvolve downloading all of the sound effects when the text source isinitially identified but that may be problematic when the computingdevice is a resource constrained device (e.g., smart speaker).

Aspects and implementation of the present technology address the aboveand other deficiencies by providing enhancements to a computing deviceto enhance its ability to conserve computing resources and still providespecial effects that are syncronized with a text source being readaloud. This may be accomplished by using data of a text source (e.g.,book) to predict future audible actions and to prefetch the physicaleffects associated with those audible actions. In one example, thetechnology may enable the computing device to predict when a user willreach a word in the text source before the word is spoken. This mayinvolve identifying effects data for a text source that correlate aphysical effect with one or more audible actions of a user. The audibleactions may include a word spoken by the user or may be a page turn,book close, or other action that produces an audible response. Thetechnology may determine a triggering condition based on a currentreading location, reading speed, other data, or a combination thereof.The triggering condition may be before the audible action arises, e.g.before the user speaks the word in the text source, turns the page,closes the book, or the like. In response to detecting that thetriggering condition is satisfied, the technology may cause thecomputing device to load content for the physical effect andsubsequently provide the physical effect to modify an environment of theuser.

Systems and methods described herein include technology that enhancesthe technical field of pre-caching based on recognition of human speech.In particular, the technology disclosed may address technical problemsassociated resource consumption when analyzing speech and downloadingspecial effects. The technology may also reduce a delay in providingspecial effects so that the special effects are better synchronized withthe human speech. The technology may also allow the download (e.g.pre-caching) of the content for the physical effect to be adaptive, e.g.in real time or near real time, to the computing resources, such asnetwork bandwidth, processing power, and/or the like, that are presentlyavailable.

The technology discussed below includes multiple enhancements to acomputing device with or without virtual assistant features. Theenhancements may be used individually or together to optimize theability of the computing device to follow along while a text source isbeing read aloud and to provide special effects to supplement anenvironment of a listening user. In one example, the environment mayinclude a parent reading a book aloud to one or more children. Inanother example, the environment may include one or more users providinga presentation, speech, or other performance to an audience. In eitherexample, the technology may be used to enhance the environment withspecial effects based on an analysis of data associated with the textsource. The special effects may be synchronized with particular portionsof the text source, such as a particular spoken word or a page turn.

FIG. 1 illustrates an example environment 100 that includes a textsource that is being read aloud and one or more devices that supplementthe environment to enhance the listening experience of a user, inaccordance with one or more aspects of the disclosure. Environment 100may be a physical environment such as an indoor setting (e.g., bedroom,conference room), outdoor setting (park, field), or other location.Environment 100 may be referred to as a ubiquitous computing environmentor pervasive computing environment and may include embedded computingfunctionality. The embedded computing functionality may provide ambientintelligence that is sensitive and responsive to the presence of humans.In one example, environment 100 may include one or more users 110A and110B, a text source 120, one or more computing devices 130A and 130B,and one or more physical effect devices 140A-C.

Users 110A and 110B may include human users that are able to perceivecontent of a text source. User 110A may be an individual user that isreading the content of the text source or may be multiple users that areeach reading a portion of one or more text sources. User 110A may bereferred to as a reader, presenter, announcer, actor, other term, or acombination thereof. User 110B may listen to content of the text sourcethat is being read aloud. User 110B may or may not read along with user110A. In one example, user 110A may be a parent that is reading to achild user 110B. In another example, user 110A may include one or morepresenters speaking to one or more users 110B that are members of anaudience. In either example, the content of text source 120 may beannounced for one or more other users to hear.

Text source 120 may be any source of content that can be interpreted andread aloud. Text source 120 may include content that contains numbers,characters, words, symbols, images, or a combination thereof. Thecontent may be arranged into a sequence that can be uttered by a userwhen read or after being memorized. Text source 120 may be a physical orelectronic book, magazine, presentation, speech, script, screenplay,memorandum, bulletin, article, blog, post, message, other arrangement oftext, or a combination thereof. In the example of FIG. 1, text source120 may be a children's book that includes a sequence of words andimages that can be read aloud to a child.

Audible actions 112A-C may be any action or combination of actions thatproduce a sound that can be detected by a user or computing device.Audible actions 112A-C may be heard, perceived, or observed by an ear ofa user or by audio sensors associated with a computing device (e.g.,microphones). As shown in FIG. 1, there may be multiple types of audibleactions and they may depend on where the sound originates from. Audibleactions 112A may be a first type of audible action that includes a vocalsound (e.g., an utterance) that may originate from a human voice or acomputer synthesized voice. The vocal sound may be a linguistic vocalsound (e.g., spoken word), a non-linguistic vocal sound (e.g., laughing,crying, coughing), other sound, or a combination thereof. Audible action112B may be a second type of audible action that includes non-vocalsounds that originate from the user or another source and may includeclapping, finger snapping, other sound, or a combination thereof.Audible action 112C may be a third type of audible action that includesnon-vocal sounds that are caused by a user interacting with an objectand may include a page turning, a book closing, a door opening/closing,an object falling, tapping on the floor, other sound, or a combinationthereof. One or more of the audible actions 112A-C may be detected byone or more of sensors 131A-C.

Sensors 131A-C may be coupled to the computing device 130A and mayenable the computing device to sense aspects of environment 100. Sensors131A-C may include one or more audio sensors (e.g., microphones),optical sensors (e.g., ambient light sensor, camera), atmospheric sensor(e.g., thermometer, barometer, hydrometer), motion sensors (e.g.,accelerometer, gyroscope, etc.), location sensors (e.g., globalpositioning system sensors (GPS)), proximity sensors, other sensingdevices, or a combination thereof. In the example shown in FIG. 1,sensor 131A may be an audio sensor, sensor 131B may be an opticalsensor, and sensor 131C may be a temperature sensor. One or more ofsensors 131A-C may be internal to computing device 130A, external tocomputing device 130A, or a combination thereof and may be coupled tocomputing device 130A via a wired or wireless connection (e.g.,Bluetooth®, WiFi®).

Computing device 130A may be any computing device that is capable ofreceiving and processing data derived from sensors 131A-C. Computingdevice 130A may function as a voice command device and provide access toan integrated virtual assistant. In one example, computing device 130Amay include a smart speaker, mobile device (e.g., phone, tablet), awearable device (e.g., smart watch), a digital media player (e.g., smartTV, micro console, set-top-box), a personal computer (e.g., laptop,desktop, workstation), home automation device, other computing device,or a combination thereof. In some implementations, computing device 130Amay also be referred to as a “user device,” “consumer device,” or“client device.” Data generated by sensors 131A-C may be received bycomputing device 130A and may be processed locally by computing device130A or may be transmitted remotely from computing device 130A toanother computing device (e.g., 130B).

Computing device 130A may include one or more components for processingthe sensor data. In the example shown in FIG. 1, computing device 130Amay include an audio analysis component 132, a text source analysiscomponent 133, a comparison component 134, a non-linear readingrecognition component 135, a physical effect determination component136, a predictive loading component 137, and an effect providingcomponent 138. In other examples, one or more of these components or oneor more features of the components may be performed by another computingdevice (e.g., computing device 130B). These components will be discussedin more detail in regards to FIGS. 2-4 and may function to detect acurrent reading location and to instruct one or more physical effectdevices 140A-C to enhance the listening experience.

Physical effect devices 140A-C may be any computing device capable ofcausing or providing a physical effect. The physical effect may beperceived via a sense of users 110A and 110B (e.g., hearing, sight,touch, smell, and taste). Each of the physical effect devices 140A-C mayproduce one or more of the physical effects and computing device 130Amay function as one or more of the physical effect device 130A-C.Physical effect devices 140A-C may provide a physical effect 145 or mayinstruct another device to provide physical effect 145. In one example,one or more of the physical effect devices 140A-C may be part of orintegrated with a home automation system or may be separate from a homeautomation system. As shown in FIG. 1, physical effect device 130A mayinclude a speaker or other device capable of causing or emitting anacoustic effect. Physical effect device 130B may include one or morelight sources (e.g., lightbulbs, pixels) or other device capable ofaltering the amount of light present in environment 100 (e.g., motorizedshades or blinds). Physical effect device 130C may include one or moredevices that can cause a haptic effect and may include a vibrationsource (e.g., massaging chair), a fan producing wind (e.g., ceiling fanor air conditioner), a heating or cooling source (e.g., thermostat),other device, or a combination thereof.

Physical effect 145 may be any modification of environment 100 that canbe perceived by a user or a computing device and may include acousticeffects, haptic effects, optical effects, other effects, or acombination thereof. An acoustic effect may be a physical effect thatrelates to sound and can be propagated via sound waves. An acousticeffect may include human or animal sounds (e.g., voices or noises),atmospheric sounds (e.g., thunder, rain, wind, or other weather sounds),musical sounds (e.g., instruments, background music, theme music),object sounds (e.g., knocking, door opening, window shutting, glassbreaking, object crashing, automobile running), other sound effects, ora combination thereof. A haptic effect may be a physical effect thatrelates to a user's sense of touch. The haptic effect may include abreeze, a vibration, a temperature change, other touch sensation, or acombination thereof. An optical effect may be a physical effect thatrelates to light and can be propagated via visible electromagneticradiation. The optical effect may include the increase or decrease inambient lighting, light flashes, animations, other change in an amountof light, or a combination thereof. The optical effect may originatefrom lamps (e.g., ceiling lamps, desk lamps), flash lights (e.g., phonelight), window coverings (e.g., blinds or shades), projectors,electronic displays, holographic displays, lasers, other light sources,or a combination thereof. Other effects may include smell or tasterelated effects (e.g., olfactory effects).

Computing device 130B may be a server that is coupled to computingdevice 130A and may be local to or remote from environment 100.Computing device 130B may include one or more computing devices (such asrackmount servers, a server computer, a personal computer, a mainframecomputer, a laptop computer, a tablet computer, a desktop computer, arouter, etc.), data stores (e.g., hard disks, memories, databases),networks, software components, and/or hardware components. In oneexample, computing device 130B may function to provide remote processingand may function as a speech processing service as discussed in moredetail in regards to FIG. 2. In another example, computing device 130Bmay provide computing device 130A with access to media items.

The media items may correspond to physical effects, text sources,profile information, speech models, instructions, other data, or acombination thereof. Example media items may include, but are notlimited to, digital sound effects, digital music, digital animations,social media information, electronic books (e-books), electronicmagazines, digital newspapers, digital audio books, digital video,digital photos, website content, electronic journals, web blogs, realsimple syndication (RSS) feeds, electronic comic books, softwareapplications, etc. In some implementations, a media item may be referredto as a content item and may be provided over the Internet and/or viacomputing device 130A (e.g., smart speaker). As used herein, “media,”“media item,” “digital media,” “digital media item,” “content,” and“content item” can include an electronic file or record that can beloaded or executed using software, firmware, or hardware configured topresent the content to one or more users in environment 100. In oneimplementation, computing device 130B may store media items using one ormore data stores and provide the media items to computing device 130Aover network 150.

Network 150 may include one or more of a private network (e.g., a localarea network (LAN), a public network (e.g., the Internet), a wide areanetwork (WAN)), a wired network (e.g., Ethernet network), a wirelessnetwork (e.g., Wi-Fi or bluetooth connection), a cellular network (e.g.,a Long Term Evolution (LTE) network), routers, hubs, switches, servercomputers, and/or a combination thereof.

In general, functions described in one implementation as being performedby computing device 130A, computing device 130B, or physical effectdevices 140A-C may be performed by one or more of the other devices inother implementations. In addition, the functionality attributed to aparticular component can be performed by different or multiplecomponents operating together. The computing device 130A and 130B mayalso be accessed as a service provided to other systems or devicesthrough appropriate application programming interfaces. Althoughimplementations of the disclosure are discussed in terms of a smartspeaker, the implementations may also incorporate one or more featuresof a cloud service or content sharing platform.

In situations in which the systems discussed herein collect personalinformation about client devices or users, or may make use of personalinformation, the users may be provided with an opportunity to controlwhether the computing devices can collect user information (e.g.,information about a user's audio input, a user's preferences, a user'scurrent location, social network, social actions, activities, orprofession), or to control whether and/or how to receive content from acomputing device that may be more relevant to the user. In addition,certain data may be treated in one or more ways before it is stored orused, so that personally identifiable information is removed. Forexample, a user's identity may be treated so that no personallyidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby the computing device.

FIG. 2-4 depict block diagrams illustrating an exemplary computingdevice 130 that can detect a reading location within a text source andsupplement the environment with physical effects to enhance a listeningexperience, in accordance with one or more aspects of the disclosure.Computing device 130 may be the same or similar to computing devices130A, computing device 130B, or a combination thereof. FIG. 2 discussesfeatures that enable computing device 130 to receive and compare audiodata of a user with data of a text source. FIG. 3 discusses featuresthat enable computing device 130 to analyze data to detect a readinglocation based on the audio data and text source data. FIG. 4 discussesfeatures that enable computing device 130 to provide physical effects tomodify the environment of one or more listeners. The components andmodules provided in FIGS. 2-4 are exemplary and more or less componentsor modules may be included without loss of generality. For example, twoor more of the components may be combined into a single component, orfeatures of a component may be divided into two or more components. Inone implementation, one or more of the components may reside ondifferent computing devices (e.g., a client device and a server device).

Referring to FIG. 2, computing device 130 may include an audio analysiscomponent 132, a text source analysis component 133, a comparisoncomponent 134, and a data store 240. Audio analysis component 132 mayreceive and access the audio data extracted from an environment while auser reads a text source aloud. In one example, audio analysis component132 may include an audio data receiving module 212 and an acousticmodeling module 214.

Audio data receiving module 212 may receive audio data 241 that includesone or more audible actions of a user. The audio data may include spokenwords, page turns, or other audible actions that are captured from anenvironment of a user. Audio data 241 may be received directly from oneor more of the sensors in the form of an audio signal or may be receivedindirectly from a data store 240 or other computing device after thesensors store the audio data 241. Audio data 241 may be in any digitalor analog format and may be accessed or received from within one or morestorage objects (e.g., files, database records), data streams (e.g.,audio stream, video stream), data signals, other data transmission orstorage protocol, or a combination thereof. Audio data 241 may be anaudio recording and may be segmented into one or more durations (e.g.,portions, chunks, or other units) before, during, or after it isanalyzed by acoustic modeling module 214.

Acoustic modeling module 214 may analyze audio data 241 using anacoustic model to identify phoneme data 243A. The acoustic model mayrepresent known relationships between audible actions and phonemes. Aphoneme may be a unit of sound and may correspond to a sound pattern ofan audible action (e.g., spoken word). A phoneme may be a linguisticunit, non-linguistic unit, other unit, or a combination thereof.Acoustic modeling module 214 may translate the audio data into phonemesthat are stored in data store 240 as phoneme data 243A.

Phoneme data 243A may include values that represent the one or morephonemes extracted from audio data 241. Phoneme data 243A may representa sequence of phonemes using a standard or proprietary notation. Thenotation may include a particular arrangement of one or more bits,bytes, symbols, or characters that represent a phoneme. In one example,the particular arrangement may include a symbol placed beside or betweenone or more delimiters. The delimiters may include slashes, brackets,pipes, parenthesis, commas, tabs, spaces, new line character, otherseparator, or a combination thereof. The phonemes may be arranged into asequence of phonemes that represent a portion of one or more audibleactions.

Text source analysis component 133 may receive and analyze data relatedto text source 120. Text source 120 may be determined in view of userinput that is text based, speech based, touch based, gesture based, orother manner of user input. For example, a user may identify text source120 by saying the name of the text source 120 (e.g., title or author ofa book), by typing and searching for the text source, by selecting adisplayed text source, other selection mechanism, or a combinationthereof. In the example shown in FIG. 2, text source analysis component133 may include a data access module 222 and a phoneme determinationmodule 224.

Data access module 222 may access data associated with text source 120and may store the accessed data as text source data 242. Data accessmodule 222 may access data from one or more sources, which may include alocal source, a remote source, or a combination thereof. A local sourcemay be storage of computing device 130 and a remote source may bestorage of a computing device that is accessible over a networkconnection. In one example, the remote source may be the same or similarto computing device 1309 (e.g., server or cloud service). The local orremote source may store data of one or media items discussed above andcomputing source may access the data. The data may then be analyzed,filtered, combined, or modified and subsequently stored as text sourcedata 242.

Text source data 242 may be any data associated with text source 120 andmay be provided by or accessible from an author, publisher, distributor,partner, a remote server, a third party service, other source, orcombination thereof. Text source data 242 may include descriptive data,textual data, phoneme data, other data, or a combination thereof. Thedescriptive data may indicate the title, summary, source (e.g., author,publisher, distributor), table of content (e.g., chapters, sections,pages), index (e.g., phrases, page indicators), other data, or acombination thereof.

The textual data may include one or more words of text source 120. Inone example, the words may be organized as a sequence of words 122 withor without one or more images 124. The textual data may be a datastructure that arranges the words in the same or similar manner as theyare read by a user (e.g., series of consecutive words). The sequence ofwords may be limited to only the words appearing in text source 120 ormay be supplemented with words or data that indicate the existence orcontent of non-textual information (e.g., illustrations, images, tables,formatting, paragraph, pages). In another example, the words may also oralternatively be arranged in an index data structure that indicatesunique words that are present in the text source 120 but are notarranged consecutively in a manner spoken by a user. Either datastructure may be supplemented with additional information that mayinclude a word location within the text source (e.g., page, line,slide), a number of occurrences, variations of the word (e.g., tense,plural), other data, or a combination thereof. In one example, textsource 120 may be a physical book and text source data 242 may includewords from a corresponding electronic book (e.g., e-book), a third partyservice, other source, or a combination thereof.

The phoneme data of text source 120 may be the same or similar tophoneme data 243B and may be a phonetic encoding of text source 120 thatis in a fauna that is the same or similar to the phoneme data derivedfrom the audio (e.g., phoneme data 243A). In the example discussedabove, phoneme data 243B of text source 120 may be included as part oftext source data 242 and be accessed by phoneme determination module224. In another example, phoneme data 243B may be absent from textsource data 242 and may be generated by phoneme determination module224.

Phoneme determination module 224 may determine the phoneme data for aparticular text source 120. This may involve phoneme determinationmodule 224 accessing existing phoneme data 243B from a remote source,generating phoneme data 243B based on textual data, or a combinationthereof. When generating phoneme data 243B, phoneme determination module224 may access and analyze textual data of text source data 242 andconvert (e.g., derive, translate, transform, encode) the textual datainto phoneme data 243B. The generated phoneme data may then beassociated with text source 120 for future use by computing device 130or by one or more other computing devices. In one example, the textualdata may include a sequence of words and the generated phoneme data mayinclude a phonetic encoding that includes sequences of phonetic valuesrepresenting the sequence of words. The same sequence of phonetic valuesmay correspond to two words that sound the same but are spelleddifferent (e.g., homophones). Likewise, a different sequence of phoneticvalues may correspond to words that sound different even though they arespelled the same (e.g., homographs).

As discussed above, phoneme data 243A and 243B may each include asequence of phonemes that are represented using a standard orproprietary notation. The notation may be referred to as a phonetictranscription or a phoneme transcription and may include a particulararrangement of phoneme values that represent a linguistic segment. Thelinguistic segment may be any discrete unit that can be identified,either physically or auditory, in a stream of speech. A phoneme valuemay include one or more symbols, characters, bytes, bits, other value,or combination thereof. In one example, a phoneme value may berepresented by one or more Unicode characters, American Standard codefor Information Interchange (ASCII) characters, other characters, or acombination thereof. A sequence of phoneme values may represent a singleword and each individual phoneme value may represent a portion of theword. For example, a first sequence of phonemes may be /θΛm/ andrepresent the spoken word “thumb” and a second sequence of phonemes mebe /dΛm/ and represent the spoken word “dumb.” In the examples discussedbelow, phoneme data 243A and 243B may include a sequence of values andeach value may represent a phoneme of a phoneme vocabulary.

The phoneme vocabulary may include the set of possible phoneme valuesfor one or more languages. The phoneme vocabulary may be an alphabeticsystem of phonetic notation and may represent qualities of speech thatare part of oral language: phones, phonemes, intonation and theseparation of words and syllables. The phoneme vocabulary may or may notalso represent additional qualities of speech and variations of speechannunciations (e.g., lisps, mispronunciations, accents, dialects). Thephoneme vocabulary may be the same or similar to a phoneme alphabet,character set, lexis, lexicon, other variation, or a combinationthereof. In one example, the phoneme vocabulary may be the based on theInternational Phonetic Alphabet (IPA). IPA symbols may be composed ofone or more elements related to letters and diacritics. For example, thesound of the English letter

t

may be transcribed in IPA with a single letter, [t], or with a letterplus diacritics, [

]. Delimiters (e.g., slashes) may be used to signal broad or phonemictranscription; thus, /t/ may be less specific than, and could refer to,either [

] or [t], depending on the context and language. In other examples, thephoneme vocabulary may be the same or similar to the Extended SpeechAssessment Methods Phonetic Alphabet (X-SAMPA), Kirshenbaum (e.g.,ASCII-IPA), other phoneme vocabulary, or a combination thereof.

Comparison component 134 may compare the audio of user 110A with thecontent of text source 120. The examples discussed below compare theaudio and the text source using their corresponding phoneme data and inthe absence of converting the audio into text using speech recognition.Other examples may also or alternatively use textual data, descriptivedata, audio data, other data, or a combination thereof. The comparisonsmay be performed by computing device 130, by a remote computing device(e.g., cloud service), or a combination thereof. In one example,comparison component 134 may select a phoneme sequence derived from theaudio and compare it with multiple phonemes sequences derived from thetext source. In another example, comparison component 134 may compare aphoneme sequence of the text source with multiple phoneme sequencesderived from the audio. In either example, the calculation of thesimilarity measurement data may be based on a phoneme edit distance.

Phoneme edit distance module 232 may quantify how similar two phonemesequences are to one another by determining a minimum number ofoperations required to convert one phoneme sequence into an exact matchof the other phoneme sequence. The operations may include anymodification of a phoneme value (e.g., symbol) within one of thesequence of phonemes. Example operations may include primitiveoperations such as phoneme removals, insertions, substitutions,transpositions, other operation, or a combination thereof. In theexample discussed above, the first sequence of phonemes may be /θΛm/ andrepresent “thumb” and a second sequence of phonemes may be /dΛm/ andrepresent “dumb.” Although the two words differ by two letters, theirphoneme edit distance is the numeric value 1 because converting thesequences to an exact match involves a substitution of a single phoneme(e.g., θ with d). In one example, the phoneme edit distance may be alinear edit distance that is the same or similar to a Levenshteindistance. The Levenshtein distance may be based on a minimum number ofremoval, insertion, or substitution operations needed to make the twophoneme sequences equal. In other examples, the phoneme edit distancemay also or alternatively include transposition or other operations. Ineither example, the phoneme edit distance may be a numeric value that isused to determine the similarity measurement data 244.

Similarity measurement module 234 may access data of phoneme editdistance module to determine the similarity or dissimilarity between theaudio and the text source. Similarity measurement module 234 may analyzedata of phoneme edit distance module to calculate the similaritymeasurement data 244. Similarity measurement data 244 may represent thesimilarity between two or more sequences of phonemes (e.g., phoneticrepresentation of words or sets of words) and may include numeric data,non-numeric data, other data, or a combination thereof. Similaritymeasurement data 244 may be based on the edit distance of one or moresequences of phonemes. In one example, the similarity measurement data244 may include the numeric value of the phoneme edit distance. Inanother example, similarity measurement data 244 may include aprobabilistic value derived from the numeric value of the phone editdistance. For example, similarity measurement data may be a percentage,ratio, or other value that is based on one or more phoneme editdistances and one or more other values. The other value may be thenumber of phonemes in the one or more phoneme sequences or in a portionof the text source.

Data store 240 may be a memory (e.g., random access memory), a cache, adrive (e.g., solid state drive, hard drive, flash drive), a databasesystem, or another type of component or device capable of storing data.Data store 240 may also include multiple storage components (e.g.,multiple drives or multiple databases) that may span one or morecomputing devices (e.g., multiple server computers).

FIG. 3 depicts a block diagram illustrating exemplary components thatenable computing device 130 to analyze data discussed above to determinea reading location or the absence of a reading location within the textsource. As discussed above, portions of the audio may not match the textsource identically because the user may add, skip, repeat, or reordercontent of the text source when reading aloud. As a result, the phonemedata derived from the audio and the phoneme data derived from the textsource may be challenging to compare and align. In the example shown inFIG. 1, computing device 130 may include a non-linear readingrecognition component 135 that enables the computing device to determinea location within the text source that best aligns with the audio data.In one example, non-linear reading recognition component 135 may includea fuzzy matching module 352, a location identification module 354, areading speed module 356, and a reading discontinuation module 358.

Fuzzy matching module 352 may enable computing device 130 to determinewhether there is a match between the audio and the text source. Thematch may be the same or similar to a probabilistic match, best match, aclosest match, or any match that may not be an exact match but satisfiesa predetermined threshold. In one example, determining a match betweenthe audio and text source may involve detecting that a fragment of audiocomprises one or more words of the text source. The match may bedetected even if the audio or text source contains other words, ismissing words, or includes variations of the words (e.g.,mispronunciation, missing plural). The match may be referred to as afuzzy match or an approximate match and may be detected using fuzzymatching logic. The fuzzy matching logic may be used to comparesequences of phoneme values and may operate at a syllable level segment,a word level segment, a phrase level segment, a sentence level segment,other segment, or a combination thereof. In one example, the fuzzymatching may perform the fuzzy matching using an audio segment that hasa predetermined length. The predetermined length may be customizable andmay be any duration (e.g., 3+ seconds) or any number of word tokens(e.g., 3-4 words). Having a predetermined length that is much smallerthan length of the text source may enhance accuracy and performance whenaccounting for non-linear reading.

Fuzzy matching module 352 may impose one or more constraints todetermine the match. In one example, detecting a match may involve usingone or more global unweighted costs. A global unweighted cost may berelated to the total number of primitive operations necessary to converta candidate sequence of phonemes (e.g., candidate pattern from textsource) to a selected sequence of phonemes (e.g., pattern from audio).In another example, detecting a match may involve specifying a number ofoperations of each type separately, while still others set a total costbut allow different weights to be assigned to different primitiveoperations. Fuzzy matching module 352 may also apply separateassignments of limits and weights to individual phoneme values in thesequence.

Location identification module 354 may access data of fuzzy matchingmodule 352 to identify a location within the text source thatcorresponds to the audible actions of the audio (e.g., spoken words). Inone example, the text source may be a book for children and the locationmay be a reading location within a sequence of words of the book. Inother examples, the location may be within a speech, presentation,script, screenplay, other text source, or a combination thereof. Ineither example, the location may be a past, current, or future readinglocation within the text source and may be stored as location data 345.The location data may be numeric or non-numeric data that identifies oneor more particular phonemes, words, paragraphs, pages, sections,chapters, tables, images, slides, other locations, or a combinationthereof.

Location identification module 354 may determine an audible actionmatches with multiple different portions of the text source. This mayoccur when the same word or phrase (e.g., sequence of phonemes) isrepeated multiple times within the text source. Location identificationmodule 354 may detect the spoken word by analyzing the phoneme data anddetect the spoken words match with multiple candidate locations withinthe text source. Location identification module 354 may select one ormore of the multiple candidate locations based on data of the fuzzymatching module 352. The location identification module 354 may furthernarrow the candidate locations by selecting a particular location basedon phoneme data of the audio that occurred before, during, or after thespoken word (e.g., expanding predetermined segment length or usingadjacent segments).

Reading speed module 356 may access and analyze location data 345 todetermine a reading speed of the user. The reading speed data may bedetermined in view of the location data, text source data, audio data,other data, or a combination thereof and may be stored as reading speeddata 346. The reading speed may be based on a portion of location data345 that identifies at least two locations in the text source. Thelocations may correspond to particular times and determining the readingspeed may be based on a quantity of words and a quantity of time betweenthe two or more locations. In one example, the quantity of words may bebased on the content of the text source and may not take into accountcontent that was added, skipped, or repeated by the user. In anotherexample, the quantity of words may be based on the content of textsource and also on content of the audio. This may be advantageousbecause the content of the audio may indicate words were added, skipped,repeated, other action, or a combination thereof. In either example,reading speed module 356 may update reading speed data to represent thereading speed of the user over one or more durations of time.

Reading discontinuation module 358 may access and analyze any datadiscussed above to detect whether the user has discontinued reading thetext source or is still reading the text source. This may be challengingbecause the user may have stopped reading the text source but isdiscussing a concept related to the text source. As a result, there maybe an overlap in the spoken words and the content of the text source.Detecting a discontinuation of reading may be important because it mayenable computing device to avoid recording a private discussion. Readingdiscontinuation module 358 may determine whether the user hasdiscontinued reading the text source by calculating one or morecorrespondence measures.

A correspondence measure may indicate the similarity or dissimilaritybetween a segment of audio and a corresponding portion of the textsource. The correspondence measure may be a probabilistic value thatindicates a probability that the segment of audio corresponds to thelocation of the text source. The probabilistic value may be a numeric ornon-numeric value and may be same or similar to a percentage, ratio,decimal, or other value, or a combination thereof. In one example, thevalue may be between 0 and 1 (e.g., 0.97), 0 and 100 (e.g., 98), orother range of values. One end of the range may indicate that thesegment of audio definitely corresponds to the location of the textsource (e.g., 1.0 or 100) and the other range may indicate that thesegment of the audio definitely does not correspond to the location ofthe text source (e.g., value of 0).

The correspondence measure may be based on or related to multiplesimilarity measurements. For example, both measurements may be used tocompare or contrast data derived from the audio (e.g., phoneme data243A) with data derived from the text source (e.g., phoneme data 243B).The similarity measurements (e.g., phoneme edit distances) may be usedto compare or contrast a written word of the text source with a spokenword and the correspondence measure may be used to compare or contrast aset of written words with a set of words spoken over a duration of time.The duration of time of the audio (e.g., segment) may be any length oftime and may include the set of words as well as one or more otheraudible actions (e.g., page turn, book close). In one example, the audioof the user may include a first duration and a second duration and thereading discontinuation module 358 may calculate one or morecorrespondence measures for the first duration and one or morecorrespondence measures for the second duration. The correspondencemeasures may be stored as correspondence measurement data 347. In otherexamples, the correspondence measure may also or alternatively take intoaccount one or more signals such as the absence of speech input for aduration of time, the absence of recognition of story text or therecognition of specific words or phrases that may indicate a stop. Thewords or phrase may include “let's stop reading,” “let's finishtomorrow,” “OK, I'm done,” “let's pause,” other phrase, or a combinationthereof.

Reading discontinuation module 358 may compare the correspondencemeasurement data 347 for each duration against one or more predeterminedthresholds. In response to the correspondence measurement data 347 forthe first duration not satisfying the threshold (e.g., above or belowthe threshold), the reading discontinuation module 358 may determine theduration of audio corresponds to the text source and that the user audiodata corresponds to a user reading the text source. In response to thecorrespondence measurement data 347 for the second duration satisfyingthe threshold (e.g., below or above the threshold), the readingdiscontinuation may determine the duration of audio does not correspondto the text source and that the user has stopped reading the textsource. In one example, determining the correspondence measure satisfiesthe threshold may indicate the audio data is absent a match with thedata of the text source or that the audio data is different from contentof the text source.

Reading discontinuation module 358 may perform one or more actions inresponse to determining that the user has discontinued reading the textsource. In one example, reading discontinuation module 358 may transmita signal to deactivate one or more microphones associated with computingdevice to avoid capturing or recording additional audio data. In anotherexample, reading discontinuation module 358 may transmit a signal tocease analyzing the audio data (e.g., comparing audio data with the dataof the text source). The latter example may record the audio but may notaccess or analyze the audio data. In yet another example, readingdiscontinuation module may cause computing device 130 to interact withthe user before, during, or after transmitting the signal. For example,the computing device may interact with the user by providing a prompt(e.g., audio, visual, or a combination thereof). The prompt may ask theuser whether to exit a storytime mode or may inform the user that thestorytime mode has been exited and may or may not enable the user tore-enable the storytime mode.

FIG. 4 depicts a block diagram illustrating exemplary components thatenable computing device 130 to provide physical effects to enhance theexperience of a user. As discussed above, the physical effects maymodify the environment and may include acoustic effects, haptic effects,optical effects, other effects, or a combination thereof. In the exampleshown, computing device 130 may include a physical effect determinationcomponent 136, a predictive loading component 137, and an effectproviding component 138.

Physical effect deterrmination component 136 enables computing device130 to identify and provide physical effects that correspond toparticular portions of the text source. In one example, physical effectdetermination component 136 may include an audible action correlationmodule 462, a contextual data module 464, and an effect selection module466.

Audible action correlation module 462 may enable computing device tocorrelate particular physical effects with particular audible actionsthat are associated with the text source. Audible action correlationmodule 462 may determine the correlation based on effects data 448 forthe text source. Effects data 448 may indicate which physical effectscorrespond to which portions of the text source. Effects data 448 maycorrelate a particular physical effect with either a particular locationin the text source, with a particular audible action of a user, aparticular triggering condition (discussed below), or a combinationthereof. The location in the text source may relate to an audible action(e.g., spoken word or page flip) or may be unrelated to an audibleaction (e.g., user looking at a graphical image). In one example,effects data 448 may identify an audible action that includes aparticular spoken word (e.g., dog) of the text source and the physicaleffect may involve initiating an acoustic effect (e.g., barking sound)corresponding to the spoken word. In another example, effects data 448may identify an audible action (e.g., a page turn) and the physicaleffect may involve modifying an existing physical effect (e.g.,readjusting the ambient sound, light, or temperature).

Effects data 448 may be accessible by computing device 130 or may becreated by computing device 130. In one example, computing device 130may access or receive effects data 448 directly or indirectly from anauthor, publisher, distributor, partner, third party service, othersource, or combination thereof. Effects data 448 may be included withintext source data 242 or may be separate from text source data 242. Inanother example, computing device 130 may create the effects data basedon text source data 242. For example, audible action correlation module462 may analyze textual data or phoneme data and identify physicaleffects that correspond to particular portions of the text source. Ineither example, effects data 448 may be stored in data store 240 forenhanced access by computing device 130.

Contextual data module 464 may enable computing device 130 to gathercontextual data 449 associated with the user. Contextual data 449 may bebased on an environment of the user and may be obtained using one ormore sensors (e.g., sensors 131A-C). Contextual data 449 may also oralternatively be based on profile data about the user, which may beaccessible to computing device 130 via direct user input or via a remotesource (e.g., network connection with a content platform or socialnetwork). In one example, contextual data 449 may include sound data(e.g., ambient sound measurements), light data (e.g., ambient lightmeasurements), time data (e.g., morning or night), calendar data (earlyappointment tomorrow), geographic location data (e.g., zip code,address, latitude/longitude), weather data (e.g., raining, lighting,thunder, windy, cloudy), user profile data (e.g., child's name, age, orgender), user audio feedback (e.g., child crying or clapping), otherdata, or a combination thereof.

Effect selection module 466 may enable computing device 130 to selectand modify physical effects based on the effects data 448, contextualdata 449, text source data 242, other data, or a combination thereof.Effect selection module 466 may be used to select a particular physicaleffect (e.g., acoustic effect) or to modify an attribute of a physicaleffect. The attribute may relate to a physical effect's intensity,timing, tone, transition (e.g., fade in/out), other feature, or acombination thereof. The intensity may be related to a magnitude of themodification to the environment and may relate the volume (e.g.,loudness) or luminance (e.g., brightness) of the physical effect. Thetiming may relate to the speed or duration of the physical effect.Computing device 130 may select the physical effect based on the word ofthe text source and may update an attribute of the physical effect basedon the contextual data. In one example, the contextual data may includesound data of an environment of the user and the physical effect may bean acoustic effect at a volume based on the sound data. In anotherexample, the contextual data may include light data of an environment ofthe user and the physical effect may be an optical effect that modifiesa luminance of a light source based on the light data (e.g., dim orbrighten a light). In yet another example, the contextual data mayinclude user profile data of a parent or a child and indicates an age ofa listener and wherein the physical effect comprises an acoustic effectselected based on the age of the user (e.g., a more playful dog bark fora young child and a more serious dog bark for an older child).

Effect selection module 466 may use the contextual data to identifytiming aspects related to the reading of the text source. For example,the time data or calendar data may be used to distinguish between thetext source being read in the evening or being read in the morning. Inthe evening, effect selection module 466 may select physical effectsthat are more calming (e.g., less stimulating) to encourage a listenerto get ready for bed. This may involve decreasing the brightness andvolume settings for the acoustic and optical effects and/or selectingeffects that have a lower tone (e.g., softer crash effects or whispersas opposed to shouting). In the morning, effect selection module 466 mayselect physical effects that are more stimulating to encourage a user toget ready for the day. This may involve increasing the brightness andvolume settings for the acoustic and optical effects. The calendar datamay also indicate whether the reading time is associated with a weekendor weekday or if there is an appointment coming up soon (e.g., later inthe day or early the next morning). Either of these may affect how fastthe user may read the text source and how long or often the physicaleffects should be provided.

Predictive loading component 137 may enable computing device 130 topredictively load content of a physical effect before it is needed.Predictive loading may speed up the ability of computing device 130 toprovide the physical effect by loading the content of the physicaleffect prior to the physical effect being initiated. Predictive loadingmay be the same or similar to prefetching, pre-caching, cacheprefetching, other concept, or a combination thereof. In one example,predictive loading component 137 may include a prediction module 472, atrigger determination module 474, and a content loading module 476.

Prediction module 472 may enable computing device 130 to predict a timethat the user will reach a particular portion of a text source. Forexample, prediction module 472 may determine a time that a word of thetext source will be spoken before the word is spoken by the user. Thepredicted time may be a time in the future and may be deter mined basedon a reading speed of the user, a reading location of the text source,other data, or a combination thereof. In one example, the time may becalculated based on a user's reading speed (e.g., words per minute,pages per minute) and the difference (e.g., number of words, paragraphs,or pages) between a current reading location and a target location inthe text source. In other examples, prediction module 472 may use apredictive model, machine learning, neural networks, or other techniqueto enhance the predictions based on current data, historical data, or acombination thereof.

Trigger determination module 474 may enable computing device 130 todetermine a triggering condition associated with a particular physicaleffect. The triggering condition may be a loading triggering conditionor an initiation triggering condition. A loading triggering conditionindicates when to start loading the content of the physical effect. Ainitiation triggering condition indicates when to start providing (e.g.,playing) the content of the physical effect. Either triggering conditionmay correspond to a particular time or a particular location within thetext source and may be based on the effects data, text source data,other data, or a combination thereof. The particular time may be anabsolute time (e.g., at 8:32:02 pm) or a relative time (e.g., 5 secondsbefore predicted time of word or page turn). The particular location maybe a location within the text source that is prior to the word that thephysical effect is to be aligned with. The particular location may be anabsolute location (e.g., word 397) or a relative location (e.g., 5 wordsbefore word “bark”).

The determination of the triggering condition may be based on one ormore factors that are related to the content, computing device, user,other aspect of the environment, or a combination thereof. The factorsrelated to the content may include the amount of content (e.g., 1 MBfile size), the location of the content (e.g., remote storage), the format of the content (e.g., downloadable files, streaming chunks, orformat that needs to be transcoded), the duration of the content (e.g.,2 second sound effect), other aspect of the content, or a combinationthereof. The factors related to the computing device may correspond tothe amount and/or availability of computing resources of computingdevice 130 or other computing device. The computing resources mayinclude connection speed (e.g., networking bandwidth), storage space(e.g., available solid state storage), processing power (e.g., CPU speedor load), other computing resource, or a combination thereof. Thefactors related to the user may include the user's reading speed,current reading location, speech clarity, other aspect, or a combinationthereof.

Trigger determination module 474 may use one or more of the factors tocalculate a duration of time to load or provide the content of thephysical effect. The duration of time related to loading the content maybe referred to as a predicted load time and may or may not include theduration of time to provide (e.g., play) the content. In one example,trigger determination module 474 may determine the duration of time toload the content of the physical effect based on the size of the contentand the network bandwidth of the computing device 130. Triggerdetermination module 474 may use the predicted load time to identify theparticular time or location of the triggering condition. In one example,the triggering condition may be set to a time that is greater than orequal to the predicted time of the audible action (e.g., spoken word)minus the predicated load time (e.g., 5 seconds). In another example,the triggering condition may be set to be a location within the textsource that is equal to or prior to a location that the physical effectis intended to align with. This may involve selecting a location in thetext source based on the predicted load time and the reading speed. Forexample, if the user reads at a speed of 120 words per minute (i.e., 2words a second) and the predicted load time is 5 seconds then thetriggering location may be 10 or more words prior to the word that thephysical effect should align with.

Content loading module 476 may enable computing device 130 to load thecontent of one or more physical effects in advance of the physicaleffect being initiated. Loading the content may involve computing device130 transmitting or receiving one or more requests and responses and mayinvolve downloading, streaming, copying, other operation, or acombination thereof. The content may include executable data (e.g.,instructions), informational data (e.g., audio files or chunks), otherdata, or a combination thereof. The content may be stored by computingdevice 130 as content data 451 in data store 240. Computing device 130may load the content of the physical effect from a local device (e.g.,data store 240), a remote device (e.g., server or cloud service), or acombination thereof.

Effect providing component 138 may enable computing device 130 toprovide the physical effect to modify the environment of a user. Effectproviding component 138 may be initiated after the content for thephysical effect is loaded and may be timed so that the physical effectis provided at a time that aligns with the audible action it is intendedto align with. In one example, effect providing component 138 mayinclude an instruction access module 482 and a physical effectinitiation module 484.

Instruction access module 482 may access instruction data associatedwith the physical effect. The instruction data may include a set of oneor more commands, operations, procedures, tasks, other instructions, ora combination thereof. The instructions may indicate the physical effectand one or more attributes for the physical effect.

Physical effect initiation module 484 may access the instruction dataand execute the instruction data to initiate the physical effect. Thephysical effect initiation module 484 may initiate an instructionbefore, during, or after detecting the initiation triggering condition(e.g., audible action) that corresponds to the physical effect. In oneexample, the text source may include a particular word and initiatingthe physical effect may be in response to detecting the audio datacomprises the word (e.g., matching phonemes). In another example,physical effect initiation module 484 may determine the initiationtriggering condition to initiate the physical effect. The process todetermine the triggering condition to initiate the physical effect maybe the same or similar as the triggering condition used to initiateloading the content of the physical effect. The instruction may causecomputing device 130 to provide the physical effect or may causecomputing device 130 to communicate with one or more physical effectdevices to provide the physical effect. In either example, computingdevice 130 may directly or indirectly cause the physical effect tomodify the environment of the user to enhance the experience of alistening user.

In one example, physical effect initiation module 484 or effectselection module 466 may use one or more confidence thresholds to selectand/or initiate a physical effect. The one or more confidence thresholdsmay be grouped into one or more confidence intervals that categorize theprobability that the audio matches a particular location of the textsource (e.g., spoken word matches word of text source). There may be anynumber of confidence interval and a first confidence interval mayindicate that there is a low probability that the audio matches the textsource location (e.g., >50%) and each successive confidence interval maybe higher (e.g., >75%, >95%, etc). The effects data that correlate thephysical effect with a location may also include a particular confidencethreshold (e.g., minimum confidence interval). For example, providing asound effect may be associated with a higher confidence interval thentransitioning a background effect. Computing device 130 may determine ifa confidence threshold is satisfied prior to selecting or initiating thephysical effect. This may involve comparing the correspondence measuredata, similarity measure data, other data associated with the fuzzymatching, or a combination thereof. In one example, a particularlocation in the text source may be associated with multiple differentphysical effects and each may correspond to a different confidenceinterval associated with the current reading location. When theconfidence interval is higher (e.g., trust the current reading locationis accurate) a particular acoustic effect may be initiated (e.g., asound effect of a single dog barking at a higher volume) and when theconfidence interval is lower (e.g., unsure current reading location isaccurate) a different acoustic effect may be initiated (e.g., backgroundnoise of multiple dogs barking at a lower volume).

FIGS. 5-8 depict flow diagrams of respective methods 500, 600, 700, and800 for enhancing the ability of a computing device to follow along as atext source is being read aloud and to provide special effects inreal-time, in accordance with one or more aspects of the presentdisclosure. Method 500 may involve using phoneme data and fuzzy matchingestimate reading progress. Method 600 may optimize the ability of thecomputing device to detect when a user has stopped reading from the textsource and is having a private discussion. Method 700 may enable thecomputing device to provide physical effects that take into account thecontext of the user and the user's environment. Method 800 may enablethe computing device to pre-cache content of the physical effects toreduce delay and better synchronize the physical effect with the audibleactions associated with the text source.

The methods of FIGS. 5-8 and each of their individual functions,routines, subroutines, or operations may be performed by one or moreprocessors of the computer device executing the method. In certainimplementations, one or more of the methods may be performed by a singlecomputing device. Alternatively, one or more of the methods may beperformed by two or more computing devices, each computing deviceexecuting one or more individual functions, routines, subroutines, oroperations of the method. For simplicity of explanation, the methods ofthis disclosure are depicted and described as a series of acts. However,acts in accordance with this disclosure can occur in various ordersand/or concurrently, and with other acts not presented and describedherein. Furthermore, not all illustrated acts may be required toimplement the methods in accordance with the disclosed subject matter.In addition, those skilled in the art will understand and appreciatethat the methods could alternatively be represented as a series ofinterrelated states via a state diagram or events. Additionally, itshould be appreciated that the methods disclosed in this specificationare capable of being stored on an article of manufacture to facilitatetransporting and transferring such methods to computing devices. Theterm “article of manufacture,” as used herein, is intended to encompassa computer program accessible from any computer-readable device orstorage media. In one implementation, the methods may be performed byone or more of the components in FIGS. 1-4.

Referring to FIG. 5, method 500 may be performed by processing devicesof a client device (e.g., smart speaker), a server device (e.g., cloudservice), other device, or a combination thereof and may begin at block502. At block 502, the processing device may determine phoneme data of atext source. The text source may include a sequence of words and thephoneme data may be a phonetic encoding of the sequence of words thatincludes one or more sequences of phonetic values. Each of the phoneticvalues may correspond to a phoneme and the sequence of phonemes maycorrespond to a spoken word. The same sequence of phonetic values maycorrespond to words that sound the same but are spelled different (e.g.,homophones) and different sequences of phonetic values may correspond towords that are spelled the same but sound different (e.g., homographs).

The processing device may access the phoneme data from a source of thetext source or may generate the phoneme data for the text source. Theprocessing device may generate the phoneme data by phonetically encodingthe sequence of words. This may involve accessing textual data of thetext source and generating (e.g., converting, transforming, deriving)the phoneme data based on the textual data. The phoneme data may then beassociated with the phoneme data for future use.

At block 504, the processing device may receive audio data comprising aspoken word associated with the text source. The audio data may includeone or more audible actions of a user and may include spoken words, pageturns, or other audible actions that are captured from an environment ofthe user. In one example, the processing device may receive the audiodata directly from one or more of the sensors in the form of an audiosignal. In another example, the processing device may receive the audiodata from a data store or another computing device. The audio data maybe in any digital or analog format and may be accessed or received viaone or more storage objects (e.g., files, database records), datastreams (e.g., audio stream, video stream), data signals, other datatransmission or storage protocol, or a combination thereof.

At block 506, the processing device may compare the phoneme data of thetext source and phoneme data of the audio data. The comparison of theaudio data and the text source may occur in the absence of convertingthe audio data to text (e.g. recognized words) using speech recognitionand may involve comparing phoneme data corresponding to the audio dataand phoneme data corresponding to the text source. The comparing mayinclude calculating a numeric value representing a similarity betweentwo or more sequences of phonetic values. The numeric value may be aphoneme edit distance between phoneme data of the audio data and phonemedata of the text source. The comparison may also involve performingfuzzy matching between phoneme data corresponding to the audio data andthe phoneme data of the text source.

At block 508, the processing device may identify a location in thesequence of words based on the comparison of the phoneme data of thetext source and the phone data of the audio. The identification of thelocation may involve determining a spoken word matches a word in thesequence of words of the text source. In one example, the text sourcemay be a book and the location may be a current reading location in thebook. Responsive to completing the operations described herein abovewith references to block 508, the method may terminate.

Referring to FIG. 6, method 600 may be performed by the same processingdevice discussed above or a different processing device and may begin atblock 602. At block 602, the processing device may receive audio datacomprising a spoken word associated with a text source. The audio datamay be segmented (e.g., tokenized, fragmented, partitioned, divided)into a first duration and a second duration. In one example, the textsource may be a book and the first portion of the audio data maycorrespond to the content of the book (e.g., include a spoken word ofthe book) and the second portion of the audio data may not correspond tothe content of the book (e.g., may be absent a spoken word from thebook).

At block 604, the processing device may compare the audio data with dataof the text source. The data of the text source may include phoneme dataand comparing the audio data and the data of the text source may involvephoneme comparisons. In one example, comparing phoneme data may involvecalculating a phoneme edit distance between the phoneme data of the textsource and phoneme data of the audio data.

At block 606, the processing device may calculate a correspondencemeasure between the second duration of the audio data and the data ofthe text source. Calculating the correspondence measure may involvecalculating the correspondence measure based on a plurality of phonemeedit distances. In one example, the processing device may select a setof spoken words (e.g., 3, 4, 5+ words) and compare the set of spokenwords to content of the text source. A phoneme edit distance may bedetermined for each word in the set or for a combination of one or moreof the words. The resulting numeric value may then be weighted,aggregated, or modified, to determine the correspondence measure.

At block 608, the processing device may transmit a signal to ceasecomparing audio data with the data of the text source responsive todetermining the correspondence measure satisfies a threshold.Determining the correspondence measure satisfies the threshold mayinvolve determining the correspondence measure is below a thresholdvalue or above a threshold value. The determination may also be based onthe duration of time that the correspondence measure satisfies or doesnot satisfy the threshold. Determining the correspondence measuresatisfies the threshold may indicate the second duration of the audiodata includes content that is different from content of the text sourceand may or may not indicate the audio data is absent any match with thedata of the text source. Transmitting the signal may involvetransmitting a signal to deactivate one or more microphones capturingaudio data. In one example, the processing device may cause a computingdevice to prompt the user to exit a storytime mode in response todetermining the second duration of the audio data is absent content ofthe text source. The prompt may be an audio prompt, a visual prompt,other prompt, or a combination thereof. Responsive to completing theoperations described herein above with references to block 608, themethod may terminate.

Referring to FIG. 7, method 700 may be performed by the same processingdevice discussed above or a different processing device and may begin atblock 702. At block 702, the processing device may receive audio datacomprising a spoken word of a user. The spoken word may be associatedwith a text source the user is reading aloud and may include one or moreother audible actions, such as page turns, spoken words not within thetext source, and other audible actions captured from an environment ofthe user. In one example, the processing device may receive the audiodata directly from one or more of the sensors in the form of an audiosignal (e.g. for use in real time or perceived real time). In anotherexample, the processing device may receive the audio data from a datastore or another computing device. The audio data may be in any digitalor analog format and may be accessed or received from within one or morestorage objects (e.g., files, database records), data streams (e.g.,audio stream, video stream), data signals, other data transmission orstorage protocol, or a combination thereof.

At block 704, the processing device may analyze contextual dataassociated with the user. The contextual data may include sound data,light data, time data, weather data, calendar data, user profile data,other data, or a combination thereof. In some examples, the contextualdata can be associated with a physical effect so that the processingdevice can provide physical effects that take into account the contextof the user and the user's environment. In one example, the contextualdata may include sound data of an environment of the user and thephysical effect may include an acoustic effect at a volume based on thesound data. In another example, the contextual data may include lightdata of an environment of the user and the physical effect may includean optical effect that modifies a luminance of a light source based onthe light data. In yet another example, the contextual data may includeuser profile data indicating an age of a child and the physical effectmay include an acoustic effect selected based on the age of the child.

At block 706, the processing device may determine a match between theaudio data and data of a text source. The processing device may identifythe text source based on user input (e.g., audio data or touch data) andretrieve the data of the text source. The data of the text source mayinclude phoneme data and determining the match may involve calculating aphoneme edit distance between the phoneme data of the text source andphoneme data of the audio data. In one example, determining the matchbetween the audio data and data of a text source may involve detectingthe audio data comprises a word of the text source using phoneme data ofthe text source.

At block 708, the processing device may initiate a physical effect inresponse to determining the match. The physical effect may correspond tothe text source and be based on the contextual data. The physical effectmay modify an environment of the user and may include at least one of anacoustic effect, an optical effect, and a haptic effect. The text sourcemay include a word and initiating the physical effect may be responsiveto detecting the audio data comprises the word. In one example, theprocessing device may select the physical effect based on the word ofthe text source and may update an attribute (e.g., volume or brightness)of the physical effect based on the contextual data. Responsive tocompleting the operations described herein above with references toblock 708, the method may terminate.

Referring to FIG. 8, method 800 may be performed by processing devicesof a server device or a client device and may begin at block 802. Atblock 802, the processing device may identify effects data for a textsource, wherein the effects data correlates a physical effect with anaudible action of a user. The effects data may indicate the physicaleffect and indicate a location in the text source that relates to theaudible action. The location may correspond to a word, paragraph, page,or other location of the text source. In one example, the audible actionmay be a spoken word of the text source and the physical effect may bean acoustic effect corresponding to the spoken word. In another example,the audible action may include a page turn and the physical effect maybe a modification of an existing acoustic effect, optical effect, orhaptic effect.

At block 804, the processing device may receive audio data comprising aplurality of audible actions. The plurality of audible actions mayinclude one or more spoken words of the text source and one or moreother audible actions, such as page turns, spoken words not within thetext source, and other audible actions captured from an environment ofthe user. In one example, the processing device may receive the audiodata directly from one or more of the sensors in the form of an audiosignal (e.g. for use in real time or near/perceived real time). Inanother example, the processing device may receive the audio data from adata store or another computing device. The audio data may be in anydigital or analog format and may be accessed or received from within oneor more storage objects (e.g., files, database records), data streams(e.g., audio stream, video stream), data signals, other datatransmission or storage protocol, or a combination thereof.

At block 806, the processing device may determine a triggering conditionbased on the effects data and the text source. In one example,determining the triggering condition may involve determining thephysical effect is associated with a first location in the text sourceand selecting a second location in the text source that is before thefirst location. The selection may be based on a reading speed and a loadtime associated with the physical effect and the second location may beassociated with at least one of a particular instance of a word, aparagraph, a page, or a chapter of the text source. The processingdevice may then set the triggering condition to correspond with thesecond location in the text source. In another example, determining thetriggering condition may involve calculating a duration of time to loadthe content based on an amount of the content for the physical effectand an amount of available computing resources. The computing resourcesmay relate to one or more of networking bandwidth, storage space, orprocessing power and the duration may be longer when the availablecomputing resources are lower. In one example, determining the time inthe future that the audible action will occur may involve identifying atime to initiate the loading based on the calculated duration and thedetermined time of the audible action and initiating the loading of thecontent at or before the identified time. In another example,determining the time comprises calculating the time in the future basedon a reading speed and a current reading location in the text source. Inyet another example, determining the time comprises predicting a time aword of the text source will be spoken before the word is spoken.

At block 808, the processing device may load content for the physicaleffect responsive to satisfying the triggering condition. The triggeringcondition may be satisfied prior to the occurrence of the audibleaction.

At block 810, the processing device may provide the physical effect tomodify an environment of the user. Responsive to completing theoperations described herein above with references to block 810, themethod may terminate.

The technology discussed herein includes multiple enhancements to acomputing device with or without virtual assistant features. The belowdiscussion includes multiple different enhancements that may be usedindividually or together to optimize the ability of the computing deviceto follow along while a text source is being read aloud and to providespecial effects to supplement an environment of a user. In one example,the environment may include a parent reading a book aloud to one or morechildren. In another example, the environment may include one or moreusers providing a presentation, speech, or other performance to anaudience. In either example, the technology may be used to enhance theenvironment with special effects based on an analysis of data associatedwith the text source. The special effects may be synchronized withparticular portions of the text source, such as a particular spoken wordor a page turn.

In a first example, an enhancement may be related to reading progressestimation based on phonetic fuzzy matching and confidence intervals andmay relate to the field of computer-based recognition of human speechand, in particular, to enhancing the ability of a computer device toidentify a reading location in a text source as a user reads the textsource aloud. Many technical problems arise when a computing deviceattempts to use traditional virtual assistant features to follow alongas a user reads a text source aloud. Some of the problems arise becausetraditional virtual assistant features perform speech recognition totranslate audio into text/recognized words. The speech recognitiontypically involves an acoustic step that translates the audio intophonemes and a language step that translates the phonemes intotext/recognized words. The language step often waits for subsequentspoken words to establish context before translating a spoken word intotext. The language step introduces an unnecessary time delay andconsumes additional computing resources. In addition, using therecognized text to perform a traditional text based comparison with thetext source may be more error prone then performing a phoneticcomparison (e.g., phoneme comparison). This often arises because manywords that sound the same or sound similar may be spelled verydifferently and would yield false negatives when textually compared. Inaddition, a traditional textual comparison may not properly account forsituations where a user may jump around when reading a text source. Forexample, portions of the text source may be skipped, repeated, or newcontent may be added. This may make it challenging to identify a currentreading location within the text source and to correctly detect areading speed. Aspects and implementations of the present technologyaddress the above and other deficiencies by providing enhancements toenable a computing device to detect a current reading location in a textsource as the text source is being read aloud. In one example, thetechnology may avoid the language step of traditional speech recognitionby comparing the phoneme data derived from the audio with phoneme dataderived from the text source. The text source may be a book, magazine,presentation, speech, script, or other source that includes a sequenceof words. The technology may receive audio data that includes the wordsspoken by a user and may convert the audio data to phoneme data locallyor with the assistance or a remote server (e.g., cloud service). Thephoneme data of the audio and the text source may then be compared via aphonetic comparison as opposed to a more traditional textual comparison.The phonetic comparison may be accompanied with fuzzy matching toidentify a location within the sequence of words (e.g., current readinglocation). Systems and methods described herein include technology thatenhances the technical field of computer-based recognition of humanspeech. In particular, the technology disclosed enhances the latency,accuracy, and computing resources required to identify a current readingposition. This may be the result of modifying the speech analysisprocess (e.g., speech recognition) to avoid translating the audio datainto text/words. The technology may use a speech analysis process thattranslates the audio into phoneme data using an acoustic model but mayavoid the language step that translates the phoneme data to text/wordsusing a language model. Avoiding the language step reduces latency andthe consumption of computing resources. Performing phoneme comparisonsand using fuzzy matching may enhance the accuracy in identifying acurrent reading position because it may better compensate for non-linearreading of the text source (e.g., skipping, repeating, or addingcontent).

In a second example, an enhancement may be related to algorithmicdetermination of a story readers discontinuation of reading and may berelated to the field of computer-based recognition of human speech and,in particular, to enhancing the ability of a computer device todetermine a user is no longer reading content of a text source aloud.Many technical problems arise when a computing device attempts to usetraditional virtual assistant features to follow along as a user reads atext source aloud. Some of the problems arise because a traditionalvirtual assistant may be unable to detect when a user has finishedproviding audio input if the user continues to talk about somethingelse. This may result in the computing device to continue recording theaudio of the user, which may be problematic if the user transitions todiscussing something private. Detecting when a user has stopped readingfrom a text source may be more challenging when a user does not followalong with the text and skips, repeats, or adds new content whilereading the text source aloud. Aspects and implementation of the presenttechnology address the above and other deficiencies by enhancing theability of a computing device to detect when a user has discontinuedreading a text source. In one example, the technology may enable thevirtual assistant to more accurately detect that the user has taken abreak from reading the text source and may deactivate a microphone toavoid capturing private audio content. This may involve receiving audiodata comprising the spoken word associated with a text source andcomparing the audio data with data of the text source. The technologymay calculate a correspondence measure between content of the audio dataand the content of the text source. The correspondence measure may beprobabilistic value that is based on a comparison of phoneme data,textual data, or other data and may involve using fuzzy matching logic.When the correspondence measure satisfies a threshold (e.g., below aminimum correspondence threshold), the technology may cause a signal tobe transmitted that will cease the analysis of subsequent audio data.Systems and methods described herein include technology that enhancesthe technical field of computer-based recognition of human speech. Inparticular, the technology may address technical problems by avoiding aninadvertent recording of a user's private conversation by usingcomparisons that better compensate for non-linear reading of the textsource (e.g., skipping, repeating, adding content). For example, theabove technology may facilitate more accurate and/or more promptautomated control of the virtual assistant to record and/or process onlypertinent audio. The technology may also enable the computing device toreduce power consumption by deactivating an audio sensor (e.g.,microphone) and associated data processing when the computing devicedetects the user has stopped reading the text. Furthermore, the abovetechnology may enable the computing device to reduce utilization ofcomputing resources, such as processing capacity, network bandwidth,data storage and/or the like, that may otherwise be used to recordand/or process the audio data once the user has stopped reading thetext.

In a third example, an enhancement may be related to dynamic adjustmentof story time special effects based on contextual data and may berelated to the field of virtual assistants and, in particular, toenhancing the ability of a virtual assistant to provide special effectswhile a text source is being read aloud. Modern computing devices may beconfigured to adopt traditional virtual assistant features to providesound effects that supplement an environment when a user is reading abook aloud. For example, when a user reads the word “bark” aloud thecomputing device may provide a barking sound effect. The sound effectsare often provided by the same entity that provided the text source andmay directly correspond with a portion of the text source. As a result,the special effects may be the same independent of the user orenvironment and may not be optimized to the particular readingenvironment of a user. Aspects and implementation of the presenttechnology address the above and other deficiencies by enabling acomputing device to provide a wide variety of special effects that arebased on the user's environment. In one example, the technology mayenable the computing device to analyze contextual data of an environmentof the user and to select or customize the special effects. The specialeffects may be physical effects that alter the environment of the userto include acoustic effects (e.g., music, sound effects music), opticaleffects (e.g., flashing lights, ambient light), haptic effects (e.g.,vibrations, wind, temperature changes), other effects, or a combinationthereof. The technology may involve receiving and analyzing contextualdata associated with the user. The contextual data may be related to theweather, lighting, time of day, user feedback, user profile, otherinformation, or a combination thereof. The technology may select ormodify a physical effect that corresponds to the text source based onthe contextual data. For example, this may result in selecting ormodifying a volume, brightness, speed, tone, or other attribute of thephysical effect. Systems and methods described herein include technologythat enhances the technical field of virtual assistants and homeautomation. In particular, the technology may enable a computing deviceto optimize an environment by using contextual data about the user andthe environment to add, remove, or modify physical effects to enhance alistening experience of the user.

In a fourth example, an enhancement may be related to the detection ofstory reader progress for pre-caching special effects and may be relatedto the field of virtual assistants and, in particular, to enhancing theability of a virtual assistant to pre-cache special effects for a textsource that is being read aloud. Many technical problems arise whenattempting to use traditional virtual assistant features to providesound effects that are syncronized with the spoken content of a textsource. Some of the problems arise because a traditional virtualassistant performs speech recognition to translate audio into text andthen performs a comparison based on the text. The speech recognitiontypically involves an acoustic step that translates the audio intophonemes and a language step that translates the phonemes into text. Thelanguage step often waits for subsequent spoken words to establishcontext before translating a spoken word into text. The language stepintroduces a time delay and consumes additional computing resources on acomputing device that may be resource constrained. The delay may befurther compounded because the sound effects may be large audio filesthat are downloaded from a remote data source. A traditional approachmay involve downloading the sound effects in response to detecting aspoken word but the delay may result in the special effect beingprovided long after the word is spoken. Another approach may involvedownloading all of the sound effects when the text source is initiallyidentified but that may be problematic when the computing device is aresource constrained device (e.g., smart speaker). Aspects andimplementation of the present technology address the above and otherdeficiencies by providing enhancements to a computing device to enhanceits ability to conserve computing resources and still provide specialeffects that are syncronized with a text source being read aloud. Thismay be accomplished by using data of a text source (e.g., book) topredict future audible actions and to prefetch the associated physicaleffects before the respective audible action arises. In one example, thetechnology may enable the computing device to predict when a user willreach a word in the text source before the word is spoken. This mayinvolve identifying effects data for a text source that correlate aphysical effect with one or more audible actions of a user. The audibleactions may include a word spoken by the user or may be a page turn,book close, or other action that produces an audible response. Thetechnology may determine a triggering condition based on a currentreading location, reading speed, other data, or a combination thereof.In response to detecting that the triggering condition is satisfied, thetechnology may cause the computing device to load content for thephysical effect and subsequently provide the physical effect to modifyan environment of the user. Systems and methods described herein includetechnology that enhances the technical field of pre-caching based onrecognition of human speech. In particular, the technology disclosed mayaddress technical problems associated with resource consumption whenanalyzing speech and downloading special effects. The technology mayalso reduce a delay in providing special effects so that the specialeffects are better synchronized with the human speech.

FIG. 9 depicts a block diagram of a computer system operating inaccordance with one or more aspects of the present disclosure. Invarious illustrative examples, computer system 900 may correspond tocomputing device 130 of FIGS. 2-4. The computer system may be includedwithin a data center that supports virtualization. In certainimplementations, computer system 900 may be connected (e.g., via anetwork, such as a Local Area Network (LAN), an intranet, an extranet,or the Internet) to other computer systems. Computer system 900 mayoperate in the capacity of a server or a client computer in aclient-server environment, or as a peer computer in a peer-to-peer ordistributed network environment. Computer system 900 may be provided bya personal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, or any device capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that device. Further, the term “computer” shallinclude any collection of computers that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methods described herein.

In a further aspect, the computer system 900 may include a processingdevice 902, a volatile memory 904 (e.g., random access memory (RAM)), anon-volatile memory 906 (e.g., read-only memory (ROM) orelectrically-erasable programmable ROM (EEPROM)), and a data storagedevice 916, which may communicate with each other via a bus 908.

Processing device 902 may be provided by one or more processors such asa general purpose processor (such as, for example, a complex instructionset computing (CISC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a microprocessor implementing other types of instructionsets, or a microprocessor implementing a combination of types ofinstruction sets) or a specialized processor (such as, for example, anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), or a networkprocessor).

Computer system 900 may further include a network interface device 922.Computer system 900 also may include a video display unit 910 (e.g., anLCD), an alphanumeric input device 912 (e.g., a keyboard), a cursorcontrol device 914 (e.g., a mouse), and a signal generation device 920.

Data storage device 916 may include a non-transitory computer-readablestorage medium 924 on which may store instructions 926 encoding any oneor more of the methods or functions described herein, includinginstructions for implementing method 500, 600, 700, or 800 and anycomponents or modules in FIGS. 1-4.

Instructions 926 may also reside, completely or partially, withinvolatile memory 904 and/or within processing device 902 during executionthereof by computer system 900, hence, volatile memory 904, andprocessing device 902 may also constitute machine-readable storagemedia.

While computer-readable storage medium 924 is shown in the illustrativeexamples as a single medium, the term “computer-readable storage medium”shall include a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of executable instructions. The term“computer-readable storage medium” shall also include any tangiblemedium that is capable of storing or encoding a set of instructions forexecution by a computer and cause the computer to perform any one ormore of the methods described herein. The term “computer-readablestorage medium” shall include, but not be limited to, solid-statememories, optical media, and magnetic media.

The methods, components, and features described herein may beimplemented by discrete hardware components or may be integrated in thefunctionality of other hardware components such as ASICS, FPGAs, DSPs orsimilar devices. In addition, the methods, components, and features maybe implemented by firmware modules or functional circuitry withinhardware resources. Further, the methods, components, and features maybe implemented in any combination of hardware resources and computerprogram components, or in computer programs.

Unless specifically stated otherwise, terms such as “initiating,”“transmitting,” “receiving,” “analyzing,” or the like, refer to actionsand processes performed or implemented by computer systems thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system registers and memories into otherdata similarly represented as physical quantities within the computersystem memories or registers or other such information storage,transmission or display devices. Also, the teens “first,” “second,”“third,” “fourth,” etc. as used herein are meant as labels todistinguish among different elements and may not have an ordinal meaningaccording to their numerical designation.

Examples described herein also relate to an apparatus for performing themethods described herein. This apparatus may be specially constructedfor performing the methods described herein, or it may comprise ageneral purpose computer system selectively programmed by a computerprogram stored in the computer system. Such a computer program may bestored in a computer-readable tangible storage medium.

The methods and illustrative examples described herein are notinherently related to any particular computer or other apparatus.Various general purpose systems may be used in accordance with theteachings described herein, or it may prove convenient to construct morespecialized apparatus to perform methods 500, 600, 700, 800 and/or eachof its individual functions, routines, subroutines, or operations.Examples of the structure for a variety of these systems are set forthin the description above.

The above description is intended to be illustrative, and notrestrictive. Although the present disclosure has been described withreferences to specific illustrative examples and implementations, itwill be recognized that the present disclosure is not limited to theexamples and implementations described. The scope of the disclosureshould be determined with reference to the following claims, along withthe full scope of equivalents to which the claims are entitled.

1. A method comprising: identifying effects data for a text source,wherein the effects data correlates a physical effect with an audibleaction of a user; receiving audio data comprising a plurality of audibleactions, wherein at least one of the plurality of audible actionscomprises a spoken word of the text source; determining, by a processingdevice, a triggering condition based on the effects data and the textsource; responsive to satisfying the triggering condition, loadingcontent for the physical effect, wherein the triggering condition issatisfied prior to the occurrence of the audible action; and providingthe physical effect to modify an environment of the user.
 2. The methodof claim 1, wherein determining the triggering condition comprises:determining the physical effect is associated with a first location inthe text source; selecting a second location in the text source that isbefore the first location, wherein the selecting is based on a readingspeed and a load time associated with the physical effect; and settingthe triggering condition to correspond with the second location in thetext source.
 3. The method of claim 2, wherein the second location inthe text source is associated with at least one of a particular instanceof a word, a paragraph, a page, or a chapter of the text source.
 4. Themethod of claim 1, wherein determining the triggering conditioncomprises: calculating a duration of time to load the content based onan amount of the content for the physical effect and an amount ofavailable computing resources; determining a time in the future that theaudible action will occur; identifying a time to initiate the loadingbased on the calculated duration and the determined time of the audibleaction; and initiating the loading of the content at or before theidentified time.
 5. The method of claim 4, wherein determining the timecomprises calculating the time in the future based on a reading speedand a current reading location in the text source.
 6. The method ofclaim 1, wherein the effects data indicates the physical effect andindicates a location in the text source that relates to the audibleaction, wherein the location indicates a word, paragraph, or a page ofthe text source.
 7. The method of claim 1, wherein the audible actioncomprises a spoken word of the text source and the physical effectcomprises an acoustic effect corresponding to the spoken word.
 8. Themethod of claim 1, wherein the audible action comprises a page turn andthe physical effect comprises a modification to an acoustic effect, anoptical effect, or a haptic effect.
 9. A system comprising a processingdevice configured to: identify effects data for a text source, whereinthe effects data correlates a physical effect with an audible action ofa user; receive audio data comprising a plurality of audible actions,wherein at least one of the plurality of audible actions comprises aspoken word of the text source; determine a triggering condition basedon the effects data and the text source; responsive to satisfying thetriggering condition, load content for the physical effect, wherein thetriggering condition is satisfied prior to the occurrence of the audibleaction; and provide the physical effect to modify an environment of theuser.
 10. The system of claim 9, wherein determining the triggeringcondition comprises: determining the physical effect is associated witha first location in the text source; selecting a second location in thetext source that is before the first location, wherein the selecting isbased on a reading speed and a load time associated with the physicaleffect; and setting the triggering condition to correspond with thesecond location in the text source.
 11. The system of claim 10, whereinthe second location in the text source is associated with at least oneof a particular instance of a word, a paragraph, a page, or a chapter ofthe text source.
 12. The system of claim 9, wherein determining thetriggering condition by the processing device comprises: calculating aduration of time to load the content based on an amount of the contentfor the physical effect and an amount of available computing resources;determining a time in the future that the audible action will occur;identifying a time to initiate the loading based on the calculatedduration and the determined time of the audible action; and initiatingthe loading of the content at or before the identified time.
 13. Thesystem of claim 12, wherein determining the time comprises calculatingthe time in the future based on a reading speed and a current readinglocation in the text source.
 14. The system of claim 9, wherein theeffects data indicates the physical effect and indicates a location inthe text source that relates to the audible action, wherein the locationindicates a word, paragraph, or a page of the text source.
 15. Thesystem of claim 9, wherein the audible action comprises a spoken word ofthe text source and the physical effect comprises an acoustic effectcorresponding to the spoken word.
 16. The system of claim 9, wherein theaudible action comprises a page turn and the physical effect comprises amodification to an acoustic effect, an optical effect, or a hapticeffect.
 17. The system of claim 9, wherein the system is configured toimplement a virtual assistant.
 18. A non-transitory computer readablemedium, which when executed by a processing device, causes theprocessing device to perform operations comprising: identifying effectsdata for a text source, wherein the effects data correlates a physicaleffect with an audible action of a user; receiving audio data comprisinga plurality of audible actions, wherein at least one of the plurality ofaudible actions comprises a spoken word of the text source; determininga triggering condition based on the effects data and the text source;responsive to satisfying the triggering condition, loading content forthe physical effect, wherein the triggering condition is satisfied priorto the occurrence of the audible action; and providing the physicaleffect to modify an environment of the user.
 19. The non-transitorycomputer readable medium of claim 18, wherein determining the triggeringcondition comprises: determining the physical effect is associated witha first location in the text source; selecting a second location in thetext source that is before the first location, wherein the selecting isbased on a reading speed and a load time associated with the physicaleffect; and setting the triggering condition to correspond with thesecond location in the text source.
 20. The non-transitory computerreadable medium of claim 19, wherein the second location in the textsource is associated with at least one of a particular instance of aword, a paragraph, a page, or a chapter of the text source.